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(54) Dynamically relocatable tlleabie displays 

(57) A plurality of tileable devices for transferring 
data, comprising a first device having a display, a proc- 
essor, and a first communication module for transferring 
data, a second device having a display, a processa and 
a second communication module for transfenring data, a 
third device having a display, a processor and a third 
communication module for transfenring data, wherein 
the first device is connected in substantially simultane- 
ous communication with the second device and the third 
device to pass data based on spatial positions of the 
respective first device, second device, and third device. 
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Description 

[0001] Reliably, quickly, artd intuitively transnnitting 
complex commands to small portable computers can be 
difficult. Small conrputer de\^ices do not generally have 
sufficient computer processing power to respond relia- 
bly to voice Of handwritten (pen based) comnnands. key- 
boards are often absent or too small for accurate finger 
input, and conventional buttons are too large or support 
too limited a command instruction set User interface 
techniques that rely on bulky external modules (full size 
infrared linked keyboards, tethered data gloves, or cam- 
era based gestural recognition equipment, for example) 
are expensive, often not readily available outside 
selected sites, and probably too awkward for wide- 
spread use in conjunction with consumer level portable 
computing devices. 

[0002] User interface designers for portable comput- 
ers have attempted to conpensate for some of these 
problems by constructing devices that rely on various 
spatial, positional, or environmental cues that manually 
or automatically allow for activation of various modes in 
the device. For example, some laptop computers use 
the action of opening/closing the lid to initiate automatic 
bootup/jpowerdown of the computer without requiring 
arty additional signal input (e.g. such as depressing a 
"start" button or typing "Lo.g.o.n" on a keyboard) from 
a user. Alternatively, tiie use of small portable comput- 
ers that automatically switch control nrrodes depending 
on position, orientation, or room location have been 
investigated. Buttonless manual control of a portable 
computer through deliberate user actions such as tilting 
the portable computer have also been described. 
[0003] However, all these solutions for interfacing with 
small portable computers have generally been limited in 
scope and functionality. What is needed is a user inter- 
face system suitable for even very small portable com- 
puters (having volumetric dimensions on the order of a 
one cubic centimeter) that is powerful, can be intuitively 
operated by an ordinary user witii little training, and is 
still readily capable of modification or extension by the 
user. The present invention meets these requirements 
by providing a manipulatory user interface tiiat 
responds to a user twisting, folding, bending, squeez- 
ing, shaking, tilting, spinning, lifting, or otherwise physi- 
cally manipulating the computer. 
[OCHM] In tiie manipulatory user interface system of 
the present invention, tiie most basic level of manipula- 
tion is known as a "senseme". A senseme is defined as 
a single indivisible type of physical manipulation. A par- 
tial list of categories of sensemes include material 
transformations such as squeezing, twisting, stretching; 
local spatial transformations such as translation, rota- 
tion, orbiting; and environmental transformations based 
on tenperature, light level, or vibration. For exanple, a 
small portable computer may support a deformable 
piece having multiple embedded sensors tiiat detect 
folding, twisting, or bending of the deformable piece by 
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a user. This conrputer can also contain a nun^er of 
accelerometers that sense relative spatial information; 
gyroscopic, radio or infrared positional sensors for 
determining absolute position; and various tiiermal or 
5 photosensors that respectively detect tenperature and 
light le/el changes. Intentional or unintentional nnodifi- 
cations detected by one or more of these sensor sys- 
tems can provide the basis for a powerful user interface 
scheme. 

10 [0005] As will be appreciated, each senseme category 
contains many individually distinguishable members. 
For example, consider the category of senseme known 
as a "pinch", a structural transformation generally com- 
pleted by a user squeezing the deformable piece 

15 between a forefinger and thumb. A pinch can be modi- 
fied by varying its speed (quick or slow pinch), magni- 
tude/intensity (light or hard pinch), portion of 
deformable piece pinched (top. bottom, or center of 
deformable piece pinched), or even portion of body 

20 used to pinch (right handed pinch or left handed pinch), 
witii each modification being distinguishable as a 
senseme capable of being mapped onto a conrputer 
control command. 

[0006] Alttiough tiie wide variety of easily distinguish- 
es able sensemes woukl alone provide a powerful user 
interface to a computer, the present invention further 
extends the flexibility of the senseme based user inter- 
face by supporting computer control based on a "mor- 
pheme" input. The morpheme is a tenporally 
30 synchronous (or overlapping asynchronous) tuple of 
one or more sensemes. Note that a morpheme can 
(and often will) contain more than one senseme. The 
sensemes combined Into a morpheme can come either 
from the same category (the user pinches witii a right 
35 hand while tapping with a left hand finger), or different 
categories (the user pinches tiie deformable piece witfi 
a right hand while modifying the spatial position of tiie 
portable conputer by tilting it fonward). 
[0007] Any morpheme can in turn be extended by par- 
40 ticipation in a "sentence". A sentence is defined as a 
sequence of one or more temporally disjoint mor- 
phemes. The sentence level allows definition of a phys- 
ical manipulatory grammar by appropriate choice of 
morpheme sequence, and corollary rules governing, for 
45 example, use of active (verb like) morphemes, naming 
(noun) morphemes, or connectors. Other possible 
grammar constructs used in sentences may include 
tiiose based on "home" systems. Home systems are 
general-purpose gestural languages, whose grammar 
£0 and syntax are not borrowed in any way from a host lan- 
guage. Examples of these languages are gestural lan- 
guages developed by deaf children of hearing parents 
who have not been exposed to American Sign Lan- 
guage (ASL), and the -plains talk" of North American 
55 Indians, which was used as a trade language. 

[0008] Accordingly, tiie present invention provides a 
metiiod for inputting information to a computer con- 
nected to a deformable piece that can be manipulated. 
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and optionally to various position sensors (both relative 
and absolute), pressure sensors, thermal sensors, or 
even light sensors. The method comprises the steps of 
manipulating the defornnable piece to provide a first 
morpheme input to the computer, with the first mor- $ 
pheme input normally triggering a first default action by 
the computer. The deformaWe piece may also be 
manipulated to provide a second morpheme input to the 
computer, with the second morpheme input converting 
the normally triggered first default action to a second to 
action. The first and second morphemes (and any sut>- 
sequent morphemes) together form a sentence that can 
be interpreted as a command to implement a computer 
controlled action, whether it be to unlock an electroni- 
cally controlled door, display a graphical image on a 75 
computer display, or begin logging on to a computer net- 
work. Advantageously, such a user interface system is 
well suited for interaction with small computer devices, 
and may even be cross-cultural to a limited extent, with 
ability to squeeze or pinch being universal human 20 
attributes. 

[0009] The present invention is particularly useful for 
portable computers that can be held in the palm of a 
hand. If the portable conputer is partially or completely 
surrounded with a defbrmable material having embed- 2s 
ded or contact pressure/deformation sensors, a user is 
able to gesture with or manipulate the whole computer 
in order to achieve some desired result. Material defor- 
mation can be implemented at various scales. For 
instance, a computer and connected deformation sen- 30 
sors mounted Inside a flexible tube could respond to 
right angle bending of tiie tube, or even complex knot- 
ting or looping of the tube. In practice however, only 
minute surface deformations are required, being just 
sufficient to provide tactile feedback to pressing, pinch- 35 
ing, or bending type manipulations. In eitfier case, how- 
ever, tiie measurement of location and pressure applied 
to a surface is sufficient to characterize the mode of 
interaction (distinguishing a pinch from a prod). 
[0010] One particularly preferred embodiment of a 40 
handheld portable computer that responds to a physical 
manipulatory grammar in accordance with the present 
invention includes a computer, a feedback module to 
provide visual, auditory, or tactile feedback to a user 
(e.g.. processor connected LCD display, audio speaker. 45 
or tactile display to present Braille or other conventional 
touch interface), and co-mounted graspatrfe, deforma- 
ble piece partially or completely sun*ounding the feed- 
back module. In addition, various thermal or pressure 
sensors are mounted to detect handedness of a user, so 
with tiie grasping hand generally being tiie non-domi- 
nant hand for the user. Depending on the handedness 
of the user (which can be considered as a morpheme), 
tiie displayed data sti'ucture is modified. For exanple, 
text displayed on an LCD display screen may be auto- ss 
matically shifted rightward on the screen to allow pen 
based annotation on the left side of the screen, tiiereby 
aiding left handed users. 



[001 1 ] Physically manipulataWe user interfaces addi- 
tionally provide an opportunity for multiple devices to 
interact in a user friendly manner. For example, a tilea- 
bte display system having multiple dsplays with touch 
sensKive deformable pieces surrounding the display 
can be used to organize documents based on relative 
position of contact of each di^lay. For example, if two 
displays initially showing separate data structures (e.g 
two different pages from two different electronic books) 
are brought togetiier in side by side touching contact, 
the displayed visual information can alter (e.g. by dis- 
playing adjacent pages of a single electronic book). As 
will be appreciated, one can also use multiple comput- 
ers connected to physically manipulataWe control ele- 
ments to construct complex commands for organizing 
data structures. 

Figure 1 is a schematic diagram of an approxi- 
mately spherical hand holdable portable computer 
having a deformable surface, a status display, and a 
pressure sensor array for detecting surface defor- 
mations; 

Figure 2 is a graphical diagram illustrating various 
physical manipulation morphemes, witii axes 
respectively illustrating complexity of senseme 
tuples required to form a morpheme, and plasticity 
of a device required to support tiie physical manip- 
ulation; 

Rgures 3-16 schematically illustrate various pre- 
ferred classes of physical manipulation mor- 
phemes; 

Figure 17 is a graphical diagram illustrating various 
spatial morphemes, with a first axis illustrating com- 
plexity of senseme tuples required to form a mor- 
pheme, and a second axis illustrating the degree of 
positional information needed to support tiie physi- 
cal manipulation (moving along a continuum from 
relative local measurement along a single specified 
dimension to absolute global measurement with six 
degrees of freedom determined); 

Rgures 18-26 schematically illustrate various pre- 
ferred spatial manipulation morphemes; 

Rgure 27 is a graphical illustration showing increas- 
ing degrees of sensor system complexity that can 
be used to detect various categories of environ- 
mental stimuli, including light effects, thermal 
effects, the electromagnetic environment, and the 
vibratory/acoustic environment; 

Rgure 28 is a graphical diagram illustrating various 
physical manipulation morphemes for multiple 
interacting devices, with axes respectively illusti-at- 
ing the complexity of senseme tuples required to 
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form a morpheme, and increasing level of physical 
contact: 

Figures 29-34 schematically illustrate various pre- 
ferred manipulation nrwrphemes for multiple inter- s 
acting devices: 

Figures 35 and 36 are schematic diagrams illustrat- 
ing "squeeze" and lilt** morphemes applicable to a 
portatjie computer: io 

Figure 37 is a schematic diagram illustrating tilt and 
squeeze morphemes used to control view of large 
two dimensional data sets with a relatively small 
display of a portable computer: is 

Figure 38 Is a schematic illustration representing a 
portable computer having a display ready to receive 
annotations from a right handed user; 

20 

Figure 39 is a schematic illustration r^resenting a 
portable computer having a display ready to receive 
annotations from a left handed user; 

Figure 40 is an electronic scheniatic illustrating ss 
components of the pressure and tilt sensrtive mod- 
ules of a portable computer such as illustrated in 
Figures 35-39. 

Figures 41 and 42 are schematic illustrations of a 30 
scanner/printer/copier using a paper shaped dis- 
play interface to support morphemic input; 

Figures 43-45 are schematic illustrations of tilable 
displays capable of supporting morphemic input: 35 

Figure 46 illustrates optical sensors and patterns 
suitable for use in conjunction with tileable displays 
such as illustrated in Figures 43-45; 

40 

Figure 47 illustrates radio transponders suitable for 
use in conjunction with tileable displays such as 
illustrated in Figures 43-45; and 

Figure 48 illustrates addressing of multiple tileable 4s 
displays. 

[0012] Figure 1 illustrates an embodiment of the 
present invention suitable for supporting a morphemic 
user interface grammar. Support of the grammar can so 
require detection of a user's physical manipulation of a 
device, detection of relative or absolute spatial location 
of tiie device, detection of various environmental factors 
acting on tiie device, and even detection and interaction 
witii multiple devices or external computer networks. As ss 
illustrated, a device 10 has a deformable surface 20 with 
an underiying deformation sensor mesh 22 for detecting 
surface deformation across or within multiple subre- 



gions of the deformable surface 20. The deformation 
sensor mesh 22 is connected to an internally contained 
processor 24 having associated memory system 26. 
For detecting various positional or environmental varia- 
bles, a sensing system 28 is also provided. The illus- 
trated device furtiier includes a feedback module 33, 
which may include an externally visible status display 30 
or a non-visual feedback module 31 (typically defivering 
auditory or tactile feedback). In the illustrated device, a 
communications system 32 for reception or transmis- 
sion of information to other electronic or computing 
devices is also provided. All these components can be 
powered by a power supply 25, which is usually an inter- 
nally mounted rechargeable battery of conventional 
construction. 

[0013] Aitiiough the device 10 is illustrated as having 
an approximately spheroidal and unitary mass, various 
other shapes are contemplated to be within the scope of 
the present invention. For example, the overall shape 
may be similar to various rectangular prisms, or can be 
ellipsoidal, toroidal, planar, or even be malleable 
enough to support a wide range of user defined in-egu- 
lar shapes. In addition, multiple cooperating sliape ele- 
ments are contemplated using conventional designs 
that permit interlocking of multiple shape elements (e.g 
using a ball and socket, a lock and key. or slidable or 
rotatable interlocked components). 
[001 4] Whatever the shape of device 10, for operation 
of tiie present invention tine device 10 is completely or 
partially enveloped by the deformable surface 20. The 
present invention supports use of a great variety of 
designs and materials for tiie deformable surface 20, 
d^ending on the required plasticity, durability, longev- 
ity, and of course, cost constraints. For example, con- 
templated designs for deformable surface 20 include, 
but are not limited to: 

a closed or open celled polymeric foam material 
having a wall thickness of millimeters to centime- 
ters, with thinner walled embodiments being sup- 
ported (e.g. by adhesive attachment) by an internal 
hard shell (constructed from polymeric or metallic 
materials), and tiiose thicker walled embodiments 
directly supporting (by, e.g. brackets or supports) 
internal components such as processor 24. Suita- 
ble foams may include those conposed in whole or 
in part of widely available synthetic rubbers such as 
polychloroprene (neoprene). polystyrenes, rubber 
or nitrile rubber latex foams, polysiloxanes. block 
polymers including styrene-butadiene or styrene 
isoprene, or any other conventional material having 
good elasticity and deformability; 
a thin single layer polymeric surface loosely 
wrapped around a internal hard shell (the hard shell 
being constructed from polymeric or metallic mate- 
rials). For example, a nylon or cotton weave, single 
layer polyethylene, syntiietic rubber (with littie or no 
foam cells present), or natural polymeric materials 
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such as leather wrapped around a polystyrene cas- 
ing can be used; 

a composite layered surface having a durable poly- 
meric outer layer supported by an inner foam layer; 
or even 

a polymeric bilayer having an intermedate fluid or 
gel layer of a viscous or thixotropic material that can 
be used to support extreme deformations. The 
intermediate layer can be relatively thick (one the 
order of centimeters), or in certain embodiments 
can have a thickness measured on micron to millim- 
eter scales. Such extremely thin layers would allow 
complex twisting, folding, curling, or crumpling 
actions, and have been described in conjunction 
with U.S. Patent 5,389.945, assigned to Xerox 
Corp.. the disclosure of which is herein specifically 
incorporated by reference. 

[0015] The deformation sensor mesh 22 can be 
embedded within, or positioned to contact the deforma- 
t)ie surfece 20. The deformation sensor mesh 22 can 
include an array of individual compressional or ten- 
sional strain sensas, or alternatively, embedded or 
attached positional sensors. For certain applications, 
continuous sensors (e.g. bilayer sheets of capacitance 
sensors) may be employed. One particularly useful con- 
tinuous sensor type uses multiple capacitance or resist- 
ance strips, with deformation pressure resulting in a 
positionally localizable analog signal proportional to the 
applied deformation pressure. Various sensor types can 
be used, including simple capacitance sensors, resis* 
tive strain sensors, analog or digital pressure switches, 
inductive sensors, or even fluid flow sensors. Depend- 
ing on the sensor type employed, sensor data can be 
directly fed to the processor 24 In digital form, or be 
transformed to digital format by an general purpose 
analog/digital converter that typically provides a 4 or 8 
bit range (although as few as one or as many as 32 bits 
may be required by various applications). An analog to 
digital converter may be internal to the processor 24 or 
provkled as an external module. As will be appreciated, 
the sensor mesh 22 is intended to include combinations 
of sensors and sensor types, which can be used over 
the whole or part of the deformable surface 20. 
[0016] A positional or environmental sensor system 
28 can also be supported by device 10. Various sensor 
modes can be supported, including absolute or relative 
positional information as determined by gyroscopic sen- 
sors, accelerometers, or acoustic or infrared ranging 
techniques. Environmental sensors, including conven- 
tional light, image, thermal, electromagnetic, vibratory 
or acoustic sensors can also be present Depending on 
the desired application, even costly environmental or 
positional sensors such as those incorporating differen- 
tial GPS positioning, image analysis or recognition, 
acoustic or voice identification, or differential thermal 
sensors can be used as morphemic input. Such mor- 
phemic input, taken in conjunction with morphemic input 



as detected by sensor mesh 22. can enhance precision 
and flexibility of a user's control of device 10. 
[001 7] As illustrated, both sensor system 28 and sen- 
sor mesh 22 are connected to the processor 24 and 

5 associated memory 26. The processor 24 and memory 
26 are typically mounted within the deformable surface 
20, by either direct attachment to the deformable sur- 
face 20 or by attachment to a hard casing positioned 
within the defonnable surface 20. Conventional CISC or 

10 RISC processors can be used in the illustrated embodi- 
ment, with low f)ower processors such as the Signetics 
87C752 or 87c751 . Motorola 68HC1 1 or 68582, or AFIM 
710 being preferred. If convenient, coprocessors such 
as analog to digital converters or digital signal proces- 

is sors can be used alone or in conjunction with a main 
processor. Conventional flash, static, or dynamic RAM 
can used in the present invention, altiiough for certain 
applications higher cost embedded DRAM may also be 
used. In some storage intensive applications, menx)ry 

20 26 can include additional harddisk storage, either 
located witiiin the device 10 or available tiirough an 
external connection. As will be appreciated, for many 
applications use of optional external communications 
can at least partially supplant use of internal processors 

25 and memory (except for that necessary to support 
required sensor or communication buffering and signal- 
ling). 

[0018] The present invention optionally supports com- 
munications with an external computer system 40 using 

30 its internal communications system 32 and associated 
transceiver 34. The external computer system 40 also 
includes a transceiver 42, a personal computer or work- 
station 44, and Is connected to a local or wide area net- 
work computer system 46. The transceivers 34 and 42 

35 can support various communication protocols and 
designs, including use of a serial tetiiered line 36 (using, 
for example the RS-232C interface protocols), use of 
infrared signals 38 adhering to widely utilized IRDA 
communication standards, or use of radiofrequency sig- 

40 nal 37 (which can be. for example, a cellular telephone, 
900MHz radio, or digital PCS telephonic communica- 
tions). Alternative communication standards, or even 
alternative communication carriers such as those based 
on optical or acoustic techniques, can of course be 

45 employed! 

[0019] As will be appreciated, in addition to direct 
communication witii external computer system 40, the 
device 10 can be directly or indirectiy maintained in con- 
tinuous or intermittent communication witii a number of 

50 suitably equipped electronic devices, including a tablet 
computer 110, or even a physically manipulatable port- 
able computer 11 similar in design and function to 
6mcB 10. Communication can be direct to a target 
device, or through an intermediary retransmitter such as 

55 computer system 40. Otiier possible communication 
targets include automation control systems, security 
authorization units, personal digital assistants, note- 
book computers, or any other suitably equipped elec- 
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tronic system. 

[0020] Results of communications with external 
devices, presentation of device stored Information, or 
device status updates can all be provided to a user 
through processor 24 controlled update of feedback s 
noodule 33. Feedback to a user can be primarily visual, 
sudi as can occur in conjunction with visual display 30. 
Generally, the display 30 can be a conventional passive 
or active matrix liquid crystal display, although use of 
more sophisticated (and expensive) displays based on io 
various electrooptical or micromechanical techniques 
can of course be used. In addition, for certain devices a 
nonimaging display such as may be formed by a small 
number of status lights (e.g. red or green LEDs), or 
localized or distributed chromatic changes (in conjunc- 75 
tion with a deformable surface 22 constructed with suit- 
able eiectrochromic materials) may be all that is 
necessary for visual feedback to the us&r. 
[0021] In some embodiments of the invention, visual 
output through display 30 may be augmented (or even 20 
replaced) with a non-visual display 31. The non-visual 
display 31 can include tactile displays based on internal 
actuators, auditory feedback, or even displays based on 
confbrmal changes in device appearance. For example, 
one possible feedback display is based on internal audi- 2s 
tory speakers (emitting a range of sounds from simple 
"beeps" to well formed speech, depending on available 
processor speed and functionality) for providing user 
feedback. As will be appreciated, non-visual display 31 
and its associated actuators or electronics can support 30 
alternative feedback modes, including, for example, 
force feedback to a user through internal actuators, tac- 
tile based feedback (e.g with multiple surface projec- 
tions for presentation of Braille or other conventional 
tactile user interface), modifications to the surface tex- 35 
ture of the device, or any other conventional mechanism 
for supplying information to a user. 
[0022] To better appreciate operation of the present 
invention, some selected modes of physical manipula- 
tion of device 10 are schematically presented in Figure 40 
1. As seen in Rgure 1. device 10 can be translationally 
moved in three spatial dimensions, as illustrated with 
reference to orthogonal force anrows 50, 51, and 52. In 
addition to translational motion, tiie device 10 can be 
rotationally moved in any or all tiiree spatial dimensions. 45 
as represented by arrows 53, 54, and 55. By use of sen- 
sor system 28 (alone or in combination with communi- 
cation system 32). the relative or absolute position and 
orientation in three dimensions can be determined. 
[0023] In addition to determining spatial position and so 
orientation through use of sensors 28. tiie device 10 can 
optionally use sensor mesh 22 to measure and localize 
transient or continuing force application, with force vec- 
tors and related timing information being determined 
and interpreted. A few possible force actions (deforma- ss 
tion modes) are schematically illustrated in Rgure 1. 
witii arrows 60 and 61 illustrating depression of surface 
20 (with the combination representing a squeeze), 
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arrows 62 and 63 illustrating sliding or rubbing deforma- 
tion (with the combination representing a twist), ard tfie 
combination of sliding arrows 65 and 66. and outward 
pull 67, together representing a pinch and outward pull. 
The strengtii of the applied force can be measured (e.g. 
a hard or soft squeeze is differentiated), its spatial 
expanse found (e.g. to differentiate between poking witii 
a fingertip or a thumb pad), and timing determined (e.g. 
to differentiate between a quick or a slow depression of 
the surface). The deformation so caused may be either 
permanent or transitory. 

[0024] As will be appreciated by those skilled in tiie 
art, each of the foregoing force actions represented by 
an-ows can be considered a senseme. Some tenrporally 
distinguishable sensemes (or combinations of sense- 
mes such as the foregoing discussed pinch/pull combi- 
nation) turther represent morphemes used as a basis 
for a morphemic grammar in accordance with the 
present Invention, All of the following described nrrar- 
phemes can be modified by numerous variations in 
applied pressure, force utilized, appendages used, body 
parts, or extraneous intermediary objects used to apply 
force. In addition, timing of various objects (whether 
quick, slow, or alternately quick and slow) can modify 
Interpretation of a morpheme. For example, if "squeeze" 
is taken as a typical morpheme, one can appreciate var- 
ious squeeze operations such as quick squeeze, slow 
squeeze, hard squeeze, soft squeeze, narrow squeeze, 
wide squeeze, squeeze between hands, squeeze 
between one hand and a user's chest or head, squeeze 
between one hand and a table or wall, a squeeze made 
between two pens or two books, or even a squeeze 
between a user's tongue and tiie roof of the mouth. For 
purposes of tiie present invention, all such squeeze 
morphemes would be considered members of tiie 
"squeeze" class, with individual variations acting as 
possible modifiers or selected cases, just as a "house 
cat" might be considered a particular member of tiie 
class of "felines", which more generally embraces lions, 
tigers, and bobcats. 

[0025] To akj in understanding the diversity of contem- 
plated physical manipulation morphemes. Figure 2 illus- 
trates selected morphemes an*anged by increasing 
plasticity of tiie device required to enable morpheme uti- 
lization, and by increasing complexity of available 
senseme tuples required to form or Interpret a mor- 
pheme applied to a particular dass of device. Beginning 
with tiie least plastic device and the simplest senseme 
set used to compose a morpheme, a definition of a pos- 
sible physical manipulation and typical function invoked 
by tiiat manipulation of a device similar (but of course 
possibly more complex) to that described In conjunction 
witii Figure 1 Is presented: 

DEPRESS 

[0026] Definition: Indenting one or more subregions of 
tiie device tfirough the application of pressure. 
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[0027] Example: As seen in Rgure 3. consider a 
device 122 having a display 123. The device 122 sup- 
ports a graphical illustration software application which 
allows users to position geometric shapes or user 
defined graphical objects. The device can have four 
pads 124 placed around It one on ea<rfi side. By 
deforming a particular side, the user indicates a desire 
to "nudge" the cun-ently selected geometric shape 125 
away from that side to a new position 1 26. 

SQUEEZE 

[0028] Definition: To deform one or more subregions 
of a device through the application of vectors of force, 
some components of which are directed towards each 
other, which compress the morphological structure of 
the device. This Is a special case of depression. 
[0029] Example: As illustrated in Rgure 4. consider a 
device 1 32 which can display one or more documents In 
either iconic (closed) or textual display (open) modes. 
By selecting an open document 135, and then squeez- 
ing a deformable edge 134 of the device 132, the user 
indicates a desire to make the document "smaller", in 
this case to iconize it as icon 136. 

FOLD 

[0030] Definition: To deform a second subregion by 
bending the first subregion such that it partially or com- 
pletely overlaps the second subregion. Further defor- 
mations can be applied to that new morphological 
structure on other subregions. 
[0031] Example: As illustrated in Figure 5, consider a 
device 1 42 which can display documents. Suppose this 
device 142 has been augmented such that a deforma- 
ble horizontal llap" 144 on a top edge of the device 142 
can be folded to partially obscure a display 143. When 
the user makes this folding gesture, the user indicates a 
desire to password-protect (TiideT the currently dis- 
played document. 

CURL 

[0032] Definition: Defonning one or mae subregions 
of the device by spirally configuring subregions relative 
to each other in a cylindrical or circular morphological 
structure. 

[0033] Example: As illustrated in Rgure 6. consider a 
device 150 which can display documents in various lan- 
guages (English, French, etc.). When the user takes 
such a device 150. curls in a direction Indicated by 
arrow 157 into a tube, and then uncurls it, this "abra- 
cadabra" gesture tells the device 150 to display the cur- 
rent document in a different language. 

STRETCH 

[0034] Definition: Deforming one or more subregions 



of a device through the application of vectors of forces, 
some conponents of which are directed away from 
each other, the vectors being applied at opposite ends 
of the device. 

5 [0035] Example: As illustrated in Rgure 7. consider a 
device 160 with a graphical software application which 
allows users to manipulate geometric shapes. By 
stretching the device 160. the user indicates a desire to 
"resize" or "rescale" the cunentiy displayed shapel 65 to 

10 a larger size 166, the amount of resizing being a func- 
tion of the amount of deformation. Note tfiat SQUEEZ- 
ING can indicate resizing the currentiy displayed to a 
smaller size. 

IS PINCH 

[0036] Definition: Manipulating one or more subre- 
gions by applying vectors of forces, aligned directiy 
towards each otiier, on opposite sides of the affected 
20 6ubregion(s). This is typically, but not exclusively, 
accomplished using two-finger tactile force. PINCH is a 
special case of SQUEEZE. 

[0037] Example: As illustrated in Rgure 8, consider a 
device 170 which can copy documents. By performing a 
25 "pinching" action 1 75, tiie user indicates that they wish 
the next set of copies to be issued in stapled form. 

DOGEAR 

30 [0038] Definition: Deforming a second subregion by 
folding a first subregion at a logical corner or edge of the 
second subregion, indicating a marker or location point 
(e.g. a bookmark) to be later referenced. DOGEAR is a 
special case of FOLD. 
35 [0039] Example: As illustrated in Figure 9. consider a 
device 1 80 which displays a subset of the pages from a 
multi-page document. By "dogearing" the upper right 
corner 185 of device 180. the user indicates that they 
wish a bookmark associated with the currentiy dis- 
40 played page or pages. 

TWIST 

[0040] Definition: Deforming one or more subregions 
45 of the device through tiie application of two opposing 
rotational forces offset from each other by a non-zero 
difference about some central axis. 
[0O41] Example: As illustrated in Rgure 10, consider 
a device 190 which, over time, becomes degraded in 
50 some aspect of its performance (Its disk becomes frag- 
mented, its memory needs gartage-collection. etc.). By 
performing a "TWISr gesture 195. the user indicates 
that they wish the device to "wring itself out", perform- 
ing, for exanple. garbage collection. 

55 

RELIEF-MAP 

[0042] Definition: Deforming one or more subregions 
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of the device by raising and or lowering them by either 
spatial transformation or the additionA-emoval of mate- 
rial. 

[0043] Exanple: As illustrated in Rgure 1 1 . consider 
a device 200 which can display documents in either 
one- or two- page format. VVhen the user "scores" the 
device by making a vertical indentation about the center 
axis of the device while it displays a single page 206. the 
device 200 interprets the morpheme to request display 
of documents in a two-page format as pages 207 and 
208. 

RIP 

[0044] Definition: Deforming one or more subregions 
of the device by introducing a spatial discontinuity, by 
applying vectors of force to partially or totally disconnect 
these subregions from the device. 
[0045] Exanple: As illustrated in Figure 12, consider 
a device 210 which can copy some or all of its informa- 
tion. When the user performs the "ripping" gesture 215, 
removing one or more subregions. the device 210 cop- 
ies its currently selected data set onto those subregions 
216 and 217. 

PERFORATE 

[0046] Definition: Deforming one or more subregions 
of the device by means of introducing a change in the 
spatial connectivity of the subregions such that a hole is 
introduced (either temporarily or permanently) in the 
device. 

[0047] Exanple: As illustrated in Figure 13, consider 
a device 220 which is used to route messages between 
various parts of a device network, and which displays 
this functionality to the user by means of lines 222 rep- 
resenting message pathways. When tiie user perforates 
the device 220 with a finger or object 224 to introduce a 
hole in one of these pathways 222. the system stops 
routing messages along that patiiway 

SIMILARITY 

[0048] Definition; Deformation of one or more subre- 
gions of a device previously configured to represent 
some other predefined object. Typically the device acts 
in a manner consistent with the behavior of the real- 
world object when it is manipulated in this manner, 
[0049] Exanple: As illustrated in Rgure 14, consider 
a device 230 which contains text-to-speech and audio 
input capability, and which presents itself to the user in 
an anatomically accurate shape of a human head. 
When the user opens lips 232 on the human head, inter- 
nal sensors detect opening of the lips and activate text- 
to-speech capability. 



3D MAP 

[0050] Definition: The morphing of a device that can 
be molded around an external object to permit a sensor 

5 mesh to determine simultaneously size and shape of 
the external object. The range of possible external 
objects Is large but is limited by the size of the device*s 
solid inner-housing and the volume of the moldable 
material in the outer housing. In this system the device 

10 has the ability to accurately sense the amount of mate- 
rial from its inner surface to the outer edge of the mold- 
able material (exanple: via ultrasonic sounding^ sinvlar 
to sonar in water) thus determining an accurate elec- 
tronic model for the shape of the molded enclosure. 

75 [0051] Exanple: As illustrated in Figure 15, by press- 
ing a device attached moldable material 244 around the 
surface of an external object (e.g. a cog 242) a device 
240 can automatically generate a CAD model of that 
object and store it in its memory. 

20 

MIMICRY 

[0052] Definition: Deforming one or more subregions 
of the device such that the resultant morphological 

25 Structure resembles a known real-world object and 
through this assodation of subregions. the device acts 
in a manner consistent with the object it resembles. 
[0053] Exanple: As illustrated in Rgure 16, consider 
a device 250 containing a computer which contains text- 

30 to-speech and audio input capability, and which 
presents itself to tiie user as a misshapen blob having 
the consistency and plasticity of moldable putty or clay 
When tiie user performs the "MIMICRY" action by mold- 
ing part of the device 250 to resemble an ear, the audio 

35 input capability is activated. 

[0054] In addition to nfK>rphemes based on physical 
manipulation, various morphemes based on varying 
degrees of relative or absolute spatial positioning are 
contemplated to be useful in practice of the present 

40 invention. To aid in understanding tiie diversity of con- 
tenrplated spatial morphemes. Figure 17 illustrates 
selected spatial morphemes an-anged by increasing 
knowledge of spatial position required to enable mor- 
pheme utilization, and by increasing complexity of avail- 

45 able senseme tuples required to form or interpret a 
morpheme applied to a particular class of device. 
Beginning with a device having only rudimentary rela- 
tive positioning functionality for supporting simple spa- 
tial sensemes. and ending with a device absolutely 

so positionable to within centimeters anywhere on Earth, a 
definition of a possible spatial manipulation and typical 
function invoked by that manipulation of a device similar 
(but of course possibly more conplex) to that described 
in conjunction witii Figure 1 is presented: 

55 

TRANSLATE (relative to device) 

[0055] Definition: The linear nnovement of a device's 
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center of mass from one position in space to another 
[0056] Example: Used to substitute fa mouse control- 
led graphical "sliders* in conventional graphical user 
interfaces. "When it is only physically possible to display 
a small amount of list large listings can still be searched 
by "scrolling- a display window in response to the 
TRANSLATE morpheme. 

SHAKE 

[0057] Definition: Spatially translating all subregions 
of a device by repeated movement in opposing direc- 
tions, such that the net translation is negligible. 
[0058] Example: As illustrated in Figure 18. consider 
a device 260 which is used as a calculating device. 
When the user performs the "SHAKE" gesture, the 
device 260 clears its accumulator. 

REVOLVE 

[0059] Definition: Rotating all subregions of a device 
by rotating the subregions about a point internal to the 
device, about any arbitrary plane. 
[0060] Example: As illustrated in Figure 19, consider 
a device 270 which displays an imag^ slice of volumet- 
ric data, such as medical data from a series of CAT 
scans. By rotating the device about a center point 272 
internal to the device 270 to a new position 274, the 
plane specifying the imaged slice Is changed accord- 
ingly 

TILT 

[0061 ] Definition: Rotating one or more subregions of 
a device by rotating the subregions such that one or 
more components of tiie rotary force are in the direction 
of gravity and the amount of rotation is between about - 
180 degrees and +180 degrees. 
[0062] Example: Consider a device which displays 
frames from an animation sequence on the side facing 
the user. As the device is tilted away, the speed of the 
animation increases - as its tilted towards the user, the 
speed of the animation decreases, analogous to opera- 
tion of a gas pedal. 

FLICK 

[0063] Definition: A fonvards TILT immediately fol- 
lowed by an opposing backwards TILT 
[0064] Example: As illustrated in Figure 20. consider 
a device 280 which can transmit some subset of its data 
to another device. When the user performs the "FLICK" 
gesture by quickly tilting in the direction of an^ow 282. 
followed by a reverse tilt along arrow 282. the device 
280 performs this transmission, towards the device (not 
shown) pointed to by the ray of the gesture. 



SPIN 

[0065] Definition: Rotating one or more subregions of 
the device by rotating the subregions about a point inter- 

5 nal to the device, such that the plane of rotation is one 
of the device's surface planes. SPIN can be considered 
a special case of REVOLVE, 
[0066] Example: As illustrated in Rgure 21 , consider 
a device 290 which can display a frame of video 295 

10 from a video sequence. The user performs the "SPIN" 
gesture in a counter-clockwise direction, the device dis- 
plays earlier frames in the sequence: when the gesture 
is performed in a clockwise direction (arrow .292), the 
device 290 displays a later frame 296 in the sequence 

15 (represented by film strip 294) . 

ORIENT 

[0067] Definition: Rotating one or more subregions of 
20 the device by rotating the subregions about the center of 
the device, such that the plane of rotation is one of the 
device's surface planes, and the amount of the rotation 
is a multiple of 90 degrees (i.e. to rotate the device 
between cardinal compass points). Orient can be con- 
25 sidered a special case of SPIN, which in turn is a spe- 
cial case of REVOLVE. 

[0068] Example: As illustrated in Rgure 22, consider 
a device 300 which can display a document in either 1 
page. 2-page, or 4-page format ("l-up". "2-up", or "4- 

$0 up"). When tiie user performs tine orient gesture in a 
clockwise direction (arrow 302). the device 300 
increases the number of document pages it is display- 
ing from one page 305 to two pages 306 and 307. Fur- 
ther orient gestures would increase the number of 

35 displayed pages. When performed in a counter-clock- 
wise direction, the device 300 decreases the nunr^er of 
pages it is displaying. 

FACE 

40 

[0069] Definition: Manipulating one or more subre- 
gions of the device such that a first set of subregions is 
no longer bottom-most and a second distinct set of sub- 
regions now assumes \he first subregions former posi- 
45 tion. 

[0070] Example: As illustrated in Rgure 23. consider 
a device 310 which displays documents, and which 
allows users to edit such documents. Consider further 
the case in which the device presents itself to the user 

50 in tiie form of a cube, in which 6 different documents are 
displayed on the 6 different faces. When the user per- 
forms the "FACE" gesture by making a particular face 
top-most, the document which is now on the top-most 
face becomes editable by tiie user, while the document 

55 which is no longer top-most is no longer editable. 
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LIFT 

[0071 ] Definition: The movement of a device*s center 
of mass in a direction opposite to the current gravitation 
force acting on the device. s 
[0072] Example: Commanding the device to display 
the computer's file system at a position one level higher 
in the hierarchy. 

PAN 10 

[0073] Definition: The application of a TRANSLATION 
to a device such that it is moved parallel to the front of 
the user's body, at a substantially constant height 
[0074] Example: Viewing a spread-sheet on a device is 
with a display so small only one cell can be shown. By 
PANing the device, the contents of the current row can 
be shown in sequence depending on the rate or amount 
of PAN. However, if the device were rotated away from 
its current orientation during the PAN, a new row would 20 
be chosen. The selection of the row could be dependent 
on the deviation from the original orientation. 

PUSH-PULL 

25 

[0075] Definition: Manipulating one or more subre- 
gions of the device by spatially translating them such 
that they are moved along a line of projection from the 
center of the device to the vertical axis of the user's 
body 30 
[0076] Example: Consider a device with audio output 
capability As the device is "pushed" further away from 
the body, its audio output level increases. 'When It's 
"pulled" towards the body, its level decreases. 

35 

WHACK 

[0077] Definition: The application of an accelerative or 
de-accelerative force to one or more subregions of a 
device such that said subregions contact or are con- 40 
tacted by an external object causing an equal and 
opposite countering force. 

[0078] Example: As illustrated in Figure 24. consider 
a device 320 which can perform long and unpredictable 
database searches. When the user performs the 45 
WHACK gesture (e.g. upon a table 322). the cun-ent 
search is aborted. 

ORIENT (relative to environment) 

so 

[0079] Definition: Manipulating two subregions of the 
device such that the line drawn between the centers of 
those two subregions alters its orientation witii respect 
to the surrounding environment. 

[0080] Example: Displaying a CAD drawing of a ss 
machine-part in 3D on the display of a mobile device. As 
the orientation of the device changes, so too does the 
viewing angle and position of the rendered image. 



ORBIT 

[0081] Definition: Rotating one or more subregions by 
rotating the subregions and/or the center of mass of the 
device about some point exterior to the physical bound- 
aries of the device, about any art>itrary axis. 
[0082] Example: As illustrat^J in Figure 25. consider 
a device 330 which can search a network database, 
such as the Worid Wide Web. for information. When the 
user performs the ^orbit" gesture, such a search is initi- 
ated. The radius 332 of the rotation 334 specifies the 
breadth of the search - wider circles specify a wider 
search. The speed of the gesture specifies the time limit 
imposed on the search - the quicker the gesture, the 
more cursory the search. 

ORBIT RELATIVE TO USER 

[0083] Definition: Rotating one or more subregions by 
rotating the subregions and/or tiie center of mass of the 
device about some point exterior to the physical bound- 
aries of the device, where said point is proximal to a 
body feature of the user. This is a special case of 
ORBIT 

[0084] Example: As illustrated in Figure 26. consider 
a device 340 which can perform audio output. By per- 
forming the ORBIT gesture (in direction indicated by 
arrow 344) about the user's ear 345, the audio output is 
activated. 

MOVEMENT IN ROOM 

[0085] Definition: The local detection of a device's 3D 
position relative to reference points found within an 
enclosing room. Differences in the measured position 
are used to trigger actions. 

[0086] Example: A virtual filing system that allows you 
to save and restore files based on the devices current 
position in the room. To save a file you might think care- 
fully about the contents of the file and then walk to the 
position in the room that might be most easily assod- 
ated with it When retrieving the file you would use the 
same thought processes and go back to the position 
you had associated with the file. On doing so the files 
associated with that position would be displayed and 
you would then be able to select the file you were look- 
ing for. The system is useful because the human mind is 
very good at remembering information that is spatially 
organized rather than in some abstract information data 
structure. 

MOVEMENT BETWEEN WIDELY SEPARATED SITES 

[0087] Definition: Manipulating one or subregions of 
the device such tiiat the detected absolute spatiar posi- 
tion of those subregions is changed. 
[0088] Example: Consider a device which can display 
information from a database of dient information. When 
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the device is moved to a different dient site, the device 
automatically updates its display to display infomnation 
for the nearest client site. 

[0089] In addition to morphemes based on physical 
manipulation or spatial positioning, various morphemes s 
based on sensed environmental conditions are contem- 
plated to be useful in practice of the present invention. 
To aid in understanding tiie diversity of contemplated 
environmental morphemes, Figure 27 illustrates 
selected environmental morphemes loosely arranged in io 
order of increasing sensor complexity needed in some 
commonly sensed environmental categories. For each 
of the proffered categories, some selected sensing sys- 
tems supportable by a device such as that described in 
conjunction with Rgure 1 are presented: is 

LIGHT 

[0090] Definition: Manipulating one or more subre- 
gions of tiie device such that the amount of light falling so 
upon those subregions changes. 
[0091] Exanrple: Consider a device used in a lecture 
hall to take notes. When the room lights are turned on. 
the light sensors detect this and turn down the backlight 
to conserve energy. When the room lights are turned off 25 
{e.g. during a slide show) the light sensors detect this 
and turn up the backlight to increase viewability. 
[0092] Light sensors can range from thresholded 
binary light detectors, to light pattern detectors, to full 
imaging systems. Advanced techniques can include 3o 
image analysis and recognition to identify objects or 
persons. 

HEAT 

35 

[0093] Definition: Manipulating one or more subre- 
gions of the device such that tiie amount of heat applied 
to those subregions changes. 
[0094] Exanple: Consider a portable computer which 
has a stylus for entering text. By looking at tine heat pro- 40 
file along the back surface of tiie computer, the compu- 
ter can detect whether it is being held witii the left hand, 
the right hand, totii hands, or neither hand, and update 
its interface accordingly 

[0095] Thermal (heat) sensors can range from simple 45 
temperature sensors to sophisticated differential ther- 
mal mappers and thermal imagers. 

ELECTROMAGNETIC 

50 

[0096] Definition: Manipulating one or more subre- 
gions of the device such tiiat tiie electromagnetic spec- 
tium applied to those subregions changes, 
[0097] Example: By analyzing the radio spectnjm. the 
device can derive estimates as to its absolute spatial 55 
position, and use that to alter its functionality. 
[0098] Electromagnetic detection can include mag- 
netic compasses, radio detection, or GPS signal detec- 



tion. More advanced techniques can include 
electromagnetic spectrum analysis and interpretation, 
such as roughly determining location based on availa- 
ble radio signals. 

VIBRATE 

[0099] Definition: Manipulating one or more subre- 
gions of the device by vibration. 
[0100] Example: Consider a device which displays 
textual information. When the user takes the device on 
the bus. the ambient vibration level sensed by the 
device changes, and the device increases ttne size of 
the displayed text to help the user compensate. 
[0101] This class of environmental morphemes can 
include detection of intermittent contacts, low frequency 
rumblings, or acoustic level detection. More advanced 
techniques requiring greater processor power include 
maximum frequency identification, spectral analysis of 
acoustic frequencies (enabling the device to distinguish 
background environmental noises from speech, for 
example), or even speech based identification of per- 
sons in the vicinity of a device. 
[0102] In addition to morphemes based on physical 
manipulation, spatial position, or sensed environmental 
factors, various morphemes based on cooperation 
between multiple interacting devices are contemplated 
to be useful in practice of tiie present invention. To aid in 
understanding the diversity of contemplated . spatial 
morphemes. Figure 28 illustrates multidevice mor- 
phemes arranged by an increasing level of possible 
physical contact, and by increasing complexity of avail- 
able senseme tuples required to form or interpret a mor- 
pheme applied to a particular class of device. Beginning 
with a device having only rudimentary edge deformation 
functionality for supporting simple spatial sensemes, 
and ending witii complex deformable or embeddable 
devices tiiat can be wrapped about each otiier, a defini- 
tion of a possible multidevice manipulation and typical 
function invoked by tiiat multidevice manipulation of 
devices similar (but of course possibly more complex) to 
tiiat described in conjunction with Figure 1 is presented: 

TOUCH 

[0103] Definition: To move one or more subregions of 
a device such that tfiey enter physical contact witii a 
subregion of a second device, in any alignment and to 
any extent Or, to take two devices so aligned and 
remove tiiat alignment. 

[0104] Example: Consider two portable conputers 
350 and 351 , tiie first of which contains a database, and 
the second of which contains an IRDA port. When the 
user touches the first computer 350 to the second conv 
puter 351, tine datal>ase Is transmitted via the second 
computer's port. 
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MATCH 

[01 05] Definition: To move one or more subregions of 
a device such that they enter physical contact with one 
or more subregions of a second device, with the subre- s 
gion(s) of the first device and the subregion(s) of the 
second device being aligned along one or more edges. 
Or, to take two devices so aligned and remove that 
alignment. 

[0106] Example: As illustrated in Figure 30. consider, io 
multiple devices 360, 361 , and 362, which contain differ- 
ent versions of the same basic database. When the 
user MATCHes the first device 360 to the second device 
361, followed by matching the third device 362 to the 
matched first and second devices, their databases are is 
reconciled (synchronized). 

STACK 

[0107] Definition: To wove one or more subregions of 20 
a device such that they enter physical contact with a 
subregion of a second device, sudi that the first device 
is now located above, but physically adjacent to. the 
second device. Or. to take two devices so aligned and 
remove (i.e. unstack) that alignment 25 
[0108] Example: As illustrated in Figure 31 , consider 
a set of devices 370, 371 . and 372, eadi of which is dis- 
playing a frame of video from a longer video sequence. 
When the devices are stacked, the ordering of the 
stacking specifies an order for the video editing, and a 30 
single composite video is now produced. 

TILE 

[01 09] Definition : To move one or more subregions of 35 
a device such that they physically contact with a subre- 
gion of a second device, such that the first device and 
second device now form a single seamless spatial unit 
or to take two devices so aligned and remove that align- 
ment. TILE is a special case of MATCH. ^ 
[01 1 0] Example: As illustrated in Rgure 32. consider 
a set of devices 380, 381, 382, 383, each of which can 
independently display a portion of a large photograph. 
When the devices are tiled, each device displays the 
portion of the photograph appropriate to its cun*ent rela- 45 
tive position in the tiled grid. 

RELATIVELY ALIGN 

[0111] Definition: To move one or more subregions of so 
a device such that they engage in a particular spatial 
relation to one or more other devices, where said 
devices are not touching. 

[0112] Example: As illustrated in Rgure 33. consider 
a set of devices 390. 391 . 392, and 393 which are dis- ss 
playing a multi-page document Whichever device is 
presently placed at the far left (device 390) displays the 
table of contents, whichever one is presently placed at 



the far right (device 393) displays the index, and the oth- 
ers display pages according to their respective loca- 
tions. As different devices can have different display 
capabilities, moving them about can alter the document 
display For example, if only one of the devices has a 
color display, when it is moved from second position to 
third position then (a) the device which was In third posi- 
tion, displaying page #2. now displays page ^^1. and (b) 
the color display, which was displaying page #1, now 
displays page #2 in color. 

WRAPPING/EMBEDDING 

[0113] Definition: Manipulating one or more subre- 
gions of a device such that these subregions spatially 
occlude or are spatially occluded by some portion of the 
second device. 

[01 1 4] Example: As illustrated in Figure 34, consider 
a first device 400 which contains the infrastnjcture to 
support the filtering of email. Consider a second set of 
devices 401 and 402 which implement particular email 
filters. When the user physically embeds device 401 (or 
402) into the first device 400. by which the first device 
400 now wraps the second device 401 (or 402). the par- 
ticular email filter supported by the second device is 
activated. 

[01 16] As those skilled in the art will appreciate, com- 
binations of any of the foregoing described morphemes 
based on physical manipulation, spatial position, envi- 
ronmental conditions, or multiple interacting devices 
can be extended by participation in a morphemic *'sen- 
tence". A sentence is defined as a sequence of one or 
more temporally disjoint morphemes. Typically, between 
about 1/1 0th of a second and 2-3 seconds suffices to 
distinguish morphemes within a sentence. Of course, in 
some circumstances and indefinite time period may 
elapse. The sentence level allows definition of a physi- 
cal manipulatory grammar by appropriate choice of a 
morpheme sequence, and corollary rules governing, for 
example, use of active (verb like) morphemes, naming 
(noun) morphemes, or connectors. Just as the position 
and relation of words in a sentence define the sen- 
tence's meaning (e.g. "horse chestnut" is not the same 
as "chestnut horse"), similarly the position and relation 
of morphemes in a manipulatory sentence define tiie 
sentence's meaning. For example, in a communication 
mode a FLICK followed by a WHACK could mean 
"transfer data and erase the local copy", while a 
WHACK followed by a FLICK means "power on the 
device and transfer data". In other settings, a FLICK or 
a WHACK could mean something entirely different. To 
better understand construction of morphemic sen- 
tences, the following examples are discussed: 

DATA TRANSFER SENTENCE 

[01 1 6] Consider a device which can transmit some or 
all of its information to another device. Furthermore, tiiis 
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transmission can be done either unencrypted, or 
encrypted (to increase security). Furthermore, thfe 
transmission of a text+graphics document can either 
include the graphics, or omit them (to save time). Sup- 
pose that the user wishes to perform the convnand 
Transmit the Information in document A. encrypted, 
omitting graphics, to machine B". Then a gestural 
sequence (morphemic sentence) to support this could 
be: 

DEPRESS - the user presses on adi^layed repre- 
sentation of A. indicating that A is to be selected for 
an upcoming operation 

FLICK- the user f Gcks the device in the direction of 
device B. Indicating that the operation is a transmis- 
sion to B 

FOLD - the user folds the top quarter of the device 
over the lower three-quarters. Indicating that the 
transmission is to be enaypted. 
TWIST - the user twists the device about its central 
axis, indicating that the data is to be Vmng out", 
I.e. the graphics are to be omitted 
SQUEEZE - the user squeezes the device, indicat- 
ing that they are confirming that the operation Is to 
proceed. 

Note that none of these gestures, in isolation, performs 
an act - the "ensemble" of temporally separated mor- 
phemes must be interpreted in order to form the com- 
plete action. 

DRAWING MODIFICATION SENTENCE 

[0117] Consider a device which displays geometric 
shapes for user manipulation. Furthermore, one of the 
manipulations supported is to resize (or rescale) a 
shape. Furthermore, suppose that this resizing can be 
done either aliased (jaggy) or anti-aliased (edges are 
snx)0thed). Suppose that the user wishes to perform 
the command "Resize shape A by 120%, about the X 
axis only, using anti-aliasing". Then a morphemic sen- 
tence to support this could be: 

DEPRESS- the user presses on a displayed repre- 
sentation of A, indicating that A Is to be selected for 
an upcoming operation 

STRETCH - the user stretches some portion of the 
device, indicating that the operation is to be a 
resize. When the user starts stretching, a portion of 
the status display displays "100", The user contin- 
ues to stretch until the status display reads "120" 
RELIEF-MAP - the user "scores" the display by 
making a horizontal line of depression. Indicating 
that the operation is to take place only about the 
horizontal (X) axis. 

DEPRESS - a circular thumb stroke is made in a 
different area of the device, Indicating that anti- 
aliasing (smoothing the edges) is to be performed. 



DATABASE PRESENTATION SENTENCE 

[01 1 8] Consider a device which contains various per- 
sonal information databases, such as a list of phone 
5 numbers, a list of addresses, and a calendar. Suppose 
that the user wishes the most appropriate of those data- 
bases displayed. Then a gestural sequence to support 
this could be: 

10 SPATIAL LOCATION - the user carries the device 
such that it is spatially proximate to either the tele- 
phone, the address book, or the refrigerator (where 
the family calendar is displayed), whichever Is 
appropriate. 

15 DEPRESS - the user touches the device to activate 
the operation. 

The device now displays the personal information 
appropriate to that location. 

20 

DATABASE RETRIEVAL SENTENCE 

[01 1 9] To extend the foregoing example of a database 
presentation sentence, conskler two computers which 
25 contain calendar databases. If the user wishes to syn- 
chronize the calendars, a suitable gestural sequence to 
support this could be: 

SQUEEZE - tiie user squeezes the device to acti- 
30 vates its gestural recognition capabilities 

ORBIT - the user orbits the device about ttie sur- 
face of the other device 3 times, indicating a desire 
to only match data for the next 3 weeks. 
MATCH - the user matches tiie edge of the device 
35 to the edge of the calendar, Indicating a desire to 
"match" contents between ttie two devices. 

PRINTER/COPIER CONTROL SENTENCE 

40 [01 20] Consider a device which can produce paper 
copies of documents. Suppose that the user wishes to 
tell such a device to produce a stapled, two sided copy 
of document A, enlarged to tiie next greater size. Then 
a gestural sequence to support this could be: 

45 

DEPRESS - ttie user presses on a displayed repre- 
sentation of A, indicating ttiat A is to be selected for 
an upcoming operation 

RIP - tiie user introduces a spatial discontinuity into 
50 a portion of tiie device, indicating ttiat the upcoming 
operation is to be a copy fcanrying away" some of 
the data). 

PINCH - ttie user pinches the upper left corner of 
ttie device, indicating that the copies are to be sta- 
55 pled. 

SQUEEZE - the user presses on ttie front and back 
of ttie device, indicating that the copy is to be two- 
sided. 
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STRETCH ' the user stretches the device, indicat- 
ing that the copy is to be an enlargement to the next 
greater size. 

FACE - the device typically has its paper emitter on 
the bottom, preventing users from making copies 5 
acddentaliy. By facing the device such that the 
emitter is on the side, the copying operation is initi- 
ated. 

LIGHT BASED CONTROL SENTENCE io 

[0121] Consider a device which can display docu- 
ments. Suppose the user is using the document while 
seated on a train, and wishes the document to display 
itself with a backlight when the train enters a tunnel, and is 
wishes the docajment to display itself in a larger font 
when the train rumbles over rough tracks. Then a ges- 
tural sequence to support this could be 

SQUEEZE - the user squeezes on the device, indi- 20 
eating that a loss of light is to be compensated for 
by a backlight. 

LIGHT - as the train enters a tunnel, the LIGHT ges- 
ture is made, and the device turns on the backlight 
WHACK - the user briskly raps the device against 25 
the palm of their hand, indicating that their desired 
preference for adjusting hard-to-read documents is 
to increase the font size. 

VIBRATE • as the train goes over a bridge, the 
vibratory gesture is sensed. Because of the posi- so 
tion of this VIBRATE morpheme (after the preced- 
ing WHACK gesture) in this morphemic sentence, 
the device now increases the font size on the dis- 
played text. 

LIGHT - the user puts the device into his suitcase. 35 
making the LIGHT gesture. In this context (with no 
SQUEEZE befae it), the LIGHT gesture causes the 
device to power-off its display 

[01 22] To better appreciate utility and construction of 40 
devices in accordance with the present invention, sev- 
eral exanples of devices are now described: 

PORTABLE COMPUTER WITH SQUEEZE AND TILT 
CONTROL 45 

[0123] A handheld portable computer 500 (e.g. a 
3Com ® PalmPilot ® ) capable of being fitted with 
deformable, pressure sensitive edging 504 is schemati- 
cally illustrated in Figures 35 and 36. The conputer 500 so 
supports a name and address software application, pro- 
viding a user viewable name-address entry field on dis- 
play 503. In this embodiment, a user can squeeze the 
deformaWe, pressure sensitive edging 504 (squeeze 
arrows 507) of the computer 500. In response, the ss 
name and address software application causes the dis- 
play 503 to animate by slowly incrementing (scrolling) 
through the name list from "A" towards "Z" entries. 



When the user squeezes edging 504 again, the soft; 
ware application stops the scrolling animation. Scrolling 
functionality is further enhanced by the use of a tilt sen- 
sor, which allows the computer's behavior to mimic con- 
ventional rotatable address books. If the computer 500 
is tilted away from the 45 degree angle at which some- 
one might typically hold it, the scrolling rate is 
increased. In this application, the closer the computer 
500 was tilted towards the user (as indicated by anrow 
506 of Fig. 36). the faster the saoll rate toward '7". 
However, if a user tilted the computer 500 bade past the 
neutral 45 degree position (as indicated by arrow 506 in 
Figure 36). the animation would move backwards with a 
velocity related to the magnitude of tilt In this way it was 
possible for a user to search for items in a long list in a 
very natural way, while only using one hand. 
[01 24] In an alternative nrxx^e schematically illustrated 
in Rgure 37. saolling speed can be completely control- 
led by pressure. The greater the squeeze pressure 
(arrows 537), the faster the list scrolls. Release of the 
applied pressure causes the scrolling to halt. In this 
alternative user interface strategy, application tilt (as 
indicated by orthogonal tilt arrows 530 and 532) could 
be used to change the direction of the scrolling through 
the list, allowing a user to search portions of a large two 
dimensional data, set (schematically illustrated as a data 
plane 520) that is not entirely visible either horizontally 
or vertically on display 503. By simply tilting the display 
503 of computer 500 as if it were a window through 
which the data plane 520 can be viewed, any particular 
portion of the data plane (for example, data subset 524) 
can be viewed. As will appreciated, in botii the foregoing 
modes tiie speed of saolling, the specific neutral tilt 
angle, and required pressures to initiate scrolling 
changes can be adjusted to fit a particular user. 

PORTABLE COMPUTER WITH HANDEDNESS 
DETECTION 

[0125] Pressure sensors have been added to aug- 
ment a conventional keyboard 551 enabled user inter- 
face to a hand holdable Windows ® CE class computer 
550 (i e- a Cassio ® Cassiopia ®) schematically illus- 
trated in Figures 38 and 39. In tiiis embodiment, user 
handedness was determined by using pressure sensors 
positioned on a right back-edge and a left back-edge of 
computer 550. User studies have found that the differ- 
ence in pressure between the right and left side gave a 
direct indication of handedness of a user. As respec- 
tively illustrated in Figures 38 and 39, handedness was 
used to justify formatted text 554 to the left (Rg. 38) or 
right (Fig. 39). tiiereby allowing more space 555 on dis- 
play 553 for an electronic annotation pen to be used to 
mark-up the text 

[01 26] For both tiie embodiment of the invention illus- 
trated by Figures 35-37. and the foregoing embodiment 
illustrated in Figures 38 and 39, material deformation of 
a spongy, elastic, or otherwise deformable material 
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must be measured. Although various techniques can be 
used to measure material deformation, including those 
based on imaging or fluid volumetric changes, one par- 
ticularly useful technique is based on the use of pres- 
sure transducers. Commercially available sensors 
measure pressure (indicative of material deformation) 
by converting a pressure change to a change in electri- 
cal characteristics. For example, inexpensive sensors 
that change resistance in response to pressure can be 
obtained in a variety of shapes and sizes, including 
paper thin sensors, and easily bendable sensor strips. 
Sensors of this kind can be customized to any particular 
shape or form that a gestural Ul might require. The 
change in resistance is usually linearly related to pres- 
sure, with the sensor generally being placed in a poten- 
tial divider network to model the pressure as a change 
in potential. For a practical circuit the resulting signal 
needs to be amplified, buffered and translated such that 
the change in value from minimum pressure to maxi- 
mum pressure spans a useful range. The modified sig- 
nal can now be fed into an analog to digital converter 
(ADC) to produce a digital representation of pressure. 
An 8-bit ADC can typically be used for most applica- 
tions, however, if greater sensitivity to pressure changes 
is required, a higher resolution ADC (e.g 16-bit ADC) 
can be used. As will be appreciated, the ADC could be 
memory mapped into tiie processor's address space as 
a peripheral, or alternatively supplied as a retrofitted 
pressure interface to existing computers capable of 
benefiting from this system. Since an RS232 connection 
is an interface that is almost universally available on 
portable computers, one strategy is to have the parallel 
output of the ADC converted into a serial RS232 frame 
using a serializer such as a UART and then level-shift 
and buffer the signal as specified by the RS232 stand- 
ard. At the computer end of the serial interface, another 
level-shifter and UART. the output of which is readable 
by the processor, performs the serial-to-parallel conver- 
sion. 

[0127] As described witfi reference to Figure 40, in 
realizing a working system a microcontroller 564 (a Sig- 
netics 87c752) with an ADC built in to combine many of 
the I/O tasks in a single chip can be used in combination 
witii level shifter 566 (a f^AX3223). This approach has 
the advantage that intelligent processing of the input 
signal Is possible with software. Tilt measurement was 
provided by a tilt sensor 567 connected to a buffer/amp 
562 to supply an analog signal to microcontroller 564. 
The pressure measurement can also be encoded within 
a protocol across the serial link. This particular micro- 
controller 564 has five ADC inputs, but by making use of 
eight digital control lines, it is possible to use only one 
ADC input and one buffer amplifier 561 , to measure up 
to eight pressure points with pressure sensors 565. This 
is achieved by using the control lines to select only one 
sensor at a time and take a reading for each using a sin- 
gle input to the ADC. After eight sensors have been 
selected, eight readings are acquired in memory The 



design is practical because the nrdaocontroller can take 
measurements and make analog to digital converaons 
at a rate far higher than is needed to communicate with 
a host conputer 569. 

5 [0128] For scrolling or handedness based software 
applications, 16 levels were determined to be adequate 
for representing a pressure measurement. In order to 
have a high data throughput to the host computer 569, 
each measurement was encoded in one byte of an 

10 RS232 frame such tiiat the four lowest bits were the 
pressure representation and tiie highest four bits were 
the sensor ID. Each frame of the RS232 data was there- 
fore completely self contained. Of course any protocol 
that limits itself to a specific number of devices in its 

IS address space will eventually have the prot>lem that 
someday applications might be designed that need to 
reference far more devices than are supported. The 
solution used in this protocol is to reserve the sensor ID 
number 15 as a special value that can extend the 

20 semantics of the encoding to include an arbitrary 
number of bytes representing a sensor or value. For the 
described software applications, the commonly used 
RS232 frame format (1 start. 8 data no parity. 1 stop bit 
at a baud rate of 9600) was chosen. 

25 [0129] In operation, the host computer 569 deter- 
mined handedness by utilizing information about tiie 
cun-ent pressure exerted on two subregions - one. on 
the back of the device, occupying roughly the left half, 
and tine otiier, also on the back of the device, occupying 

30 tiie right half. The current pressure values were con- 
verted from analog to digital form, such that a digital 
value of 0 (zero) represents no pressure, and a digital 
value of. tor example. 15 represents maximum pres- 
sure. The detection circuitry then proceeds: 

35 

if (left_sensor is high AND right.sensor is high) 
tiien 

conclude that the user is gripping the device 
40 witii both hands 

else if (left_sensor is high AND right.sensor is low) 
then 

45 conclude that the user is gripping the device 

witfi ttie left hand 

only 

50 else if (left_sensor is low AND right_sensor is high) 
tiien 

conclude the user is gripping the device with 
the right hand only 

55 

else if (left.sensor is low AND right_sensor is low) 
then 
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conclude that the user is gripping the device 
with neither hand 

Also, to optimize communication, pressure values are 
only sent when they change. In order to account for jitter 
and error in the pressure sensors, a sensor is only con- 
sidered tiigh* if its value is higher than some minimum 
threshold (e.g. '2* on the 0 to 15 range). 

SCANNER/PRINTER/COPIER SUPPORTING EDGE 
DEFORMABLE DISPLAY 

[0130] A scanner/printer/copier device 570 Is sche- 
matically illustrated in Figure 41. As seen in Figure 41 
(and in vnore detail in Rgure 42) the device 570 sup- 
ports a display 574 having a deformatjie edge 572. with 
the combination roughly shaped like a piece of paper. In 
operation, a user can place a written document in the 
device 570 for scanning. An electronic version of the 
scanned document is displayed (i.e. as text 575 in Fig- 
ure 42) on the display 574. By outwardly pulling the 
deformatjie edge 572 as indicated by arrow 577, a user 
can instruct the device 570 to resize the document 
before printing or copying. Pinching opposing sides of 
the deformaWe edge 572 {arrows 578) can further 
instruct the device 570 to print double sided copies. As 
will be appredated, various other morphemes can be 
used to provide a simple interface for interacting with 
device 570. 

TILEABLE AND STACKABLE PORTABLE DISPLAYS 

[0131] Multiple autonomous display tiles having an 
onboard display controller and at least one surface con- 
sisting substantially of a conventional display are partic- 
ularly useful for the practice of various aspects of the 
present invention. Such tiles can be interconnected in 
response to various morphemes such as TOUCH, 
FLICK. RELATIVELY ALIGN, or WHACK, or even can 
be operated without substantial user mediated morphe- 
mic input in certain situations. 
[01 32] Advantageously, tile positioning can be used as 
an interface specifier in and of itself. For example, each 
display tile can contain a video segment in independent 
memory Shuffling or reorganizing the tiles can allow 
users to physically manipulate the sequence of video 
segments to affect a physically manipulatable video 
editing system. Using the card analogy, tiles can be 
used to re-sequence documents, pages within a docu- 
ment, audio annotations, voice mail, or other temporal 
media contained within the tiles. The resultant 
sequence can then be played as a whole by using the 
tiled structure as a unit. 

[0133] For purposes of the present invention, display 
tile array configurations can be categorized as follows: 
[01 34] Close-packed display tile array 600 (Figure 43) 
in which tiles 602 are arranged to span a surface 610 in 
closely abutting, but not overlapping relationship, such 



that the continuous display area is maximized (i.e. 
seamlessly). As will be appreciated, the tiles could 
themselves form a freestanding continuous surface, or 
they could be disposed on a table or other suitable sup- 

5 port. Each of tiie tiles 602 supports a display 604 sized 
to substantially cover a front surface of each tile 602. In 
certain embodiments, a back surface of each tile 602 
can also support a display Advantageously, tiiis would 
allow for creation of free standing di^lays with images 

70 visible on tiie front and back. The surface 610 can be a 
plane, a sphere, or any artatrary shape that permits til- 
ing. 

[01 35] Loose-packed display tile array 620 (each tile 
622 having a display 624 as seen in Rgure 44) in which 

75 tiles 622 are slotted into a lattice pattern (indicated by 
dotted lines 625). Each tile can be considered to be sit- 
uated in a regular bounded lattice slot (namely, the lat- 
tice slot tiiat contains a defined center of each of the 
tiles) within the lattice where the dimensions of tine 

20 bounded slot are no more than a few times the maxi- 
mum dimension of the tile, and there is no more tiian 
one tile In any slot. Within the lattice regions any of the 
tiles can be positioned arbitratriiy and still retain tiie 
same inter-relationship with the group. It Is still posable 

25 for tiles to touch one another at tiie boundaries of two or 
more lattice slots, however, tiiis is not a requirement. 
[01 36] Free-format display tiles 630 (each tile 632 hav- 
ing a display 634 as seen in Figure 45) are similar to 
loose-packed display tiles 620 where tfie size and 

30 shape of the lattice slots may vary freely (e.g. tiie 
dimension of a lattice slot may be many times the maxi- 
mum dimension of any of tiie participating tiles). The 
only constraint on the arrangement is that there must be 
no ambiguous relationships about ttie relative connec- 

35 tivity of the each of the tiles 632. That is to say. a tile that 
is to display the next piece of information to one side of 
another tile must be unique and not be confused with 
tiie task of another tile in tiie tiling lattice. 
[01 37] 3D display tiles (packable display tiles) are cre- 

40 ated by extending the three foregoing display tile cate- 
gories. However for close packed tiling of display tiles, 
packed devices in tiie center of a 3D structure would not 
be available as the user interface. This may not matter 
as tiie surface of tfie 3D shape will expose a area that 

45 will have unique affordances for some classes of appli- 
cation. For instance, cubic tiles packed into tiie shape of 
a large cube can display, using the six faces of tiie larger 
cube, ttie various projections that could be rendered by 
viewing a 3D CAD drawing from each degree of free- 

50 dom. 

[0138] As will be appreciated, display tiles need not be 
recti-linear but can be hexagonal, circular, or of arbitrary 
shape and size. The size of tiles need not be constant 
for all tiles within a larger structure. Tiles need not be 
55 aligned but may require proximity to indicate adjoining 
points or edges. 

(01 39] Tiles need not be in physical contact to define 
tfiey are joined in a group activity but instead tiiis may 
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be a programmed function. The connectivity can also be 
discovered through a radio network from a coordinating 
server or from a distributed algorithm that draws in as 
many computers as necessary for a task, using the 
wireless network to negotiate the resources it needs. 
The tiled computers may also be connected by wired 
networking systen^. although in cases where the topol- 
ogy might need to rapidly change, this is not as desira- 
ble as a wireless system. An example of a wired 
networking system of this type is a system that uses the 
internet to include many computers in a single task, one 
in which each computer knows its relative position even 
though they may be separated by rooms or in the 
extreme cases buildings, cities or countries. However, in 
general the most useful case of display tiling is when the 
tiles are proximate enough that they can all be viewed 
by one person to create a display medium whereby the 
viewing experience is an enhancement over using a sin- 
gle display. Accordingly tiles can operate as either a 
single larger contiguous structure or they can retain 
individual properties and independent functiorts or a 
combination of both. (e.g., jumbo-tron like function. 
12x12 small displays as used in a TV studio , or picture 
in-picture features found in commercial TVs or editing 
suites). 

[0140] Depending on the type of packing, various 
schemes can be used to allow for permanent, intermit- 
tent, or even one-time communication between display 
tiles. For example, close packed tiling can make use of 
wired connectivity between conputers or could use a 
variety of wireless or optical communication technolo- 
gies. In the case of wired connectivity, edge mounted 
conventional plug and socket connectors may be used 
to create a rigid tiled array Plug and socket systems 
lend themselves to parallel connections for bulk and 
high-speed data transfer. They also provide a conven- 
ient method of power distribution, which can allow for 
one of the tiles providing a power source tiiat supports 
the rest of the array A plug and socket connection 
between tiled computers 692 in a tiled computer array 
690 (with both data and power transfer shown) is sche- 
matically illustrated by lines 695 in Figure 48. 
[0141] The exact design and locations of electrical 
connectors depends on the intended use. and subse- 
quentiy , the shape of tile components. For applications 
where large seams between displays are acceptable , 
simple rigid connectors could be attached to the center 
of each edge, providing connectivity to all the surround- 
ing tiles. Other applications might require a more com- 
plex design. For exarrple. applications which require 
high-quality information display on a seamless array 
surface (i.e. tiled blueprints), might employ spring- 
loaded contacts on the four edge connectors. The 
spring mechanism allows all array connections to be 
made below the display surface, while tiles can still be 
inserted and removed from the interior of the array 
Removal of tiles could be triggered by some host signal 
which actuates a release of the spring and pops the tile 



out of the array 

[0142] Serial connectivity can also be used in a dose 
packed arrangement such as that illustrated in Rgure 
43. It has the advantage that fewer connections have to 

5 be made and in practice it might be more reliable. How- 
ever the net bandwidth between tiles will be less than a 
parallel system. Serial communication lends itself to 
optical and wireless systems thus removing tiie need for 
any physical connections. For optical technologies 

10 alignment of the transmitter and receiver is still impor- 
tant although the clever use of light pipes and lens cap- 
ture techniques can introduce more flexibility. Wireless 
systems can use many different bands of the EM spec- 
trum (kHz. MHz, GHz), utilizing a variety of modulation 

IS techniques (amplitude modulated, frequency modu- 
lated, or those based on code division multiple access) 
and operate at a range of transmitter powers. There is 
no longer a need for direct alignment if the system is 
designed with suitable communication tolerances. The 

20 transmitter range plays a crucial part in the design. If 
there is only enough power In an emitted signal to be 
picked up within a few millimeters of a tile edge, then the 
signals will be isolated, the topology will be defined by 
the physical connectivity and the complexity of design- 

25 ing tiie system to avoid interference from neightraring 
signal sources will be minimized. However, an alterna- 
tive design is to use more powerful radios. In this case 
all tiles will be able to contact all otiier tiles and inter-tile 
connectivity needs to be defined by another parameter. 

30 Signal strength can be used or, more deliberately, infor- 
mation tiiat relates the ID of a tile to a spatial map (per- 
haps held in one master tile) describing the position of 
all tiles in the tile array In this system it is also neces- 
sary to minimize inter-tile interference. For digital 

35 packet-data systems that operate at the same fre- 
quency, can'ier sense multiple access (CSMA-CD or 
CSMA-CA) systems are well known techniques to solve 
this problem, Otiier solutions involve tiles using different 
frequencies, with frequencies reused depending on the 

40 power of tiie transmitters. This is the technique used by 
traditional cellular telephones. Yet anotiner approach is 
to use Code Division Multiple Access (CDMA) that 
relies on the overlaying of signals in the same region of 
the EM spectrum, a technique known as spread spee- 
ds trum modulation. 

[0143] For loose packed display tiles, the wireless 
techniques described above in connection with dose 
packed tiling generally become essentia! to the imple- 
mentation. However, a special case of loose packing 

50 exists in which each edge of a tile does make contact 
with every other surrounding tile, except that it may only 
be a single point of contact and not at a accurately 
defined place. A wired version of this system can be 
built in which the entire edge of each tile is a serial con- 

55 nection including one of the two vertices that define the 
edge. Communication in each direction can be achieved 
by a variety of commercially available techniques, 
including use of a one-wire interface (plus a ground) for 
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bidirection communication between tags and a reader. 
Note that the ground connection for a tile arrangement 
can be derived by sharing a common ground connec- 
tion through the surface the tiles are laid out on. For 
exanrple. the surface could be made of a metal sheet. 5 
The system may be further enhanced by ensuring that 
the edge contact is made of a magnetic material and the 
vertices have a magnet embedded in the end. Such an 
arrangement ensures that there will be a good electrical 
contact between the transmitter and the receiver. jo 
[0144] The receiver can also derive power from an 
electrically transmitted signal by bridge-rectifying it and 
storing the collected charge in a capacitor for use by its 
own electronics- Thus, power distiibution can also be 
included in a one-wire interface. In this way flexible con- is 
nectivity can be achieved to support the rapid and con- 
venient rearrangement of tiles in which only the 
minimum amount of care needs to be taken in setting up 
connectivity. 

[0145] Loosely packed tile displays such as illustrated 20 
in Rgure 45 may require tiiat the display surfaces use a 
best effort algorithm to present a unified display witii all 
the sections of the displayed image bearing the con-ect 
spatial arrangement to each other even though, in the 
case of rectangular tiles, tiiey might not be registered 25 
vertically or horizontally and have an offset angle rela- 
tive to each other. In order to implement tiie desired tile 
display algorithm, not only is the relative arrangement of 
tiles important but the exact offset (distance and angle) 
from each other is also important. There are several 30 
methods that are suitable for automatically determining 
offsets between loosely packed tiles. For example, as 
illustrated with respect to Figure 46 (showing communi- 
cating tiles 652 and 654, witii respective displays 651 
and 653). optical encoding 660 along an edge can be 35 
used to identify tile orientation. It is possible to use a 
binary coded optical pattern along an edge which is reg- 
ular and encodes the distance from a vertex at any 
point. A tile tiiat abuts, or is relatively aligned, can read 
this pattern using optical sensors 658 and 659 and 40 
determine tfie display offset in the direction of tiie edge. 
Alternatively, as illustrated by Figure 47 (showing com- 
municating tiles 672 and 674), it is possible to use radio 
based techniques that rely on signal strength triangula- 
tion. Each vertex 675 or 676 of a tile 672 can contain a 4S 
wireless transmitter and a receiver. If these vertices 
transmit a short characteristic wireless signal at well 
known times, a nearby tile 674 can use the receivers sit- 
uated at its own vertices 680. 681, 682, and 683 to tri- 
angulate the position of each transmitting vertex relative so 
to them by measuring the relative delay by which the 
signal was received. After two of the vertices of the orig- 
inal transmitting tile 672 have signaled, the adjoining tile 
674 can determine its exact position and orientation 
witiiin tfneir local region of a tile an"ay. The transmitting ss 
and receiving tiles can now swap roles, witii the result 
that both tiles will know tineir relative position. This proc- 
ess can be extended across the tile array. 



[01 46] Free format tiling dffers from loose-packed tn- 
ing in that there are no proximity, or regular format con- 
straints. To demonstrate how a free format tiling system 
might v/ork. the following example is described. Imagine 
a number of laptop computers each fitted with a Global 
Positioning System (GPS) and a radio modenx Each 
laptop can determine its position (with acceptable enror) 
and communicate it to all the other laptops by contact- 
ing them through the radio modem. After some period of 
time the laptops will all know their relative location and 
absolute location. If any computers changes its location, 
it can update the local neighbors to ensure there is 
enough understanding in the array to consider the com- 
puters are in a known tiling configuration even though 
they are not close to each other and in fact they may be 
in different geographic regiore. An application that 
might use this free format tiling system is one that 
wishes to ensure Information is being conrectly sent in a 
uniform and dispersed fashion across a very large area. 
For example, the information that each laptop computer 
receives could be instructions to release an amount of a 
pesticide used to control an insect that does damage to 
commercial crops. If the pesticide is released in too high 
a concentration in any locality, it may be hazardous to 
human health. The tiled approach allows the computers 
to roam (e.g. in the back of a truck), displaying Informa- 
tion as to the type and concentration of pesticide that is 
to be released given tiieir current relative proximity. 
[01 47] Various conventional algorithms can be used to 
support disfribution of information between autonomous 
tiled displays. These algorrthms assume a system by 
which there is a master controller generating data to be 
displayed. There is also a large array of tiled computers 
ttiat tiie system will use to display visual data and/or 
process information. Each computer in the tiled array 
contains a unique ID. It is the job of tiie master to split 
tiie data into pieces that each computer can display and 
for tills information to be packaged along with the ID of 
tine target computer. The algorithms below describe 
how the information travels from the master to a desti- 
nation display tile: 

Daisy chain routing 

[01 48] Display tiles are arranged to have a logical con- 
nectivity with each other so that each one only transmits 
information to the next in a predefined line. The comput- 
ers are said to be daisy chained to each. Any informa- 
tion sent to the start of the chain contains an ID and tfie 
first conputer in the chain compares it to its own ID. If it 
matches, it acts on the data. If not, it sends the data on 
to the next computer in the chain until it finds its destina- 
tion. 

N-ary routing 

[0149] In N-ary routing tiie path to the destination is 
contained in tiie ID of the device. A simple routing 
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example is schematically Illustrated with refrence to the 
direction arrows 695 in Rgure 48. which shows quater- 
nary routing in physically connected tileable displays. In 
quaternary routing, an array is conceptually arranged as 
a quaternary tree with each node having an input and 5 
three outputs. In this system each pair of bits of the ID 
contains a routing command, A 0 indicates send the 
packet to the first output a 1 indicates the second out- 
put, a 2 indicates the thinj output, and 3 indicates no fur- 
ther transmission. There is also a count that gets w 
decremented by each node to tell successive nodes the 
bit number that is currently being considered and when 
the packet has reached its destination. In this way pack- 
ets are forwarded from node to node with a simple 
choice at each stage until they reach their target display. 15 
As will be appreciated, it is possible to design N-ary sys- 
tems with more than three outputs (a power of two is 
usually convenient for implementation (e.g. 4, 8, 16 ...)). 

Rooding 20 

[0150] Flooding has no predefined routing structure. 
The computer that takes the first packet from the master 
checks to see if it has the correct ID. If not, the packet 
sent out on all links to which it has not yet sent or 25 
received that packet. The result Is a flood of copies of 
the packet across the array eventually reaching its des- 
tination. The packets must also have a maximum hop 
count to ensure they eventually are removed from the 
system. The disadvantage of this approach is that many 30 
more tiles are burdened with the processing of 
unwanted data than in the previous two schemes, which 
may impact the overall efficiency of the system. 

Hot Potato 35 

[0151] The Hot Potato algorithm is similar to the 
Rooding algorithm, except a retransmitted packet is 
sent out on only one output that is either chosen ran- 
domly or is the least busy The process stops when the 40 
packet reaches the correct tile. The time that a packet 
takes to reach its destination is not deterministic. 

Claims 

45 

1 . A plurality of tileable devices for transferring data, 
comprising 

a first device having a display a processor, and 
a first communication module for transferring so 
data, 

a second device having a display a processor 
and a second communication module for trans- 
ferring data, 

a third device having a display a processor and 55 
a third communication module for transferring 
data, 

wherein the first device Is connected in sub- 



stantially simultaneous comnnunication with the 
second device and the third device to pass data 
based on spatial positions of the respective first 
device, second device, and 

third device. 

2. The plurality of tileable devices of claim 1 . wherein 
the first device maintains information about spatial 
position of the second device and the third device, 
and transfers differing data to the second device 
and tiie tiiird device based on that maintained spa- 
tial position information. 

3. The plurality of tileable devices of claim 1; wherein 
the first, second and tiiird communication modules 
respectively support optical links. 

4. The plurality of tileable devices of claim 1 . wherein 
the first second and third communication modules 
respectively support radio links. 

5. The plurality of tileable devices of claim 1 , wherein 
the first second and third communication modules 
respectively support physical electrical connec- 
tions. 

6. The plurality of tileable devices of claim 1. each 
first, second and third device respectively furtfier 
comprising a first, second, and tiiird deformable 
piece, with intercommunication between at least 
two of tiie devices being triggered by a touching 
action between devices. 

7. The plurality of tileable devices of claim 1, each 
first, second and third device respectively further 
comprising a first, second, and tiiird deformable 
piece, witii intercommunication between at least 
two of the devices being triggered by a stacking 
action between devices. 

8. The plurality of tileable devices of claim 1, each 
first, second and third device respectively further 
comprising a first, second, and tiiird deformable 
piece, witii intercommunication between at least 
two of the devices being triggered by a tiling action 
between devices. 

9. A dynamic anray of devices, the an-ay conrprising. 

a plurality of devices, each device having a dis- 
play maintainable in a user determined rela- 
tionship to each other, and each device having 
a determinable information state, 
a plurality of position detectors, each device 
having at least one of the plurality of position 
detectors to determine its position relative to 
one or more of tfie plurality of devices, 
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wherein dete-minaWe information state of each 
• of the plurality of devices is modified upon its 
change of position relative to one or more 
devices. 

5 

10. A tileable display system, comprising 

a first device having a display, a processor, and 
a first communication module for transferring 
data. '^^ 
a second device having a display, a processor 
and a second communication module for trans- 
ferring data. 

wherein the first device is connected in sub- 
stantially simultaneous comnuinication with the is 
second device to pass data based on spatial 
positions of the respective first device and sec- 
ond device. 
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Fig. 4 
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Fig. 12 
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Fig. 15 




Fig. 16 
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Fig. 18 
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Fig. 19 
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Fig. 30 
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Fig. 31 




Fig. 32 
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Fig. 37 
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Fig. 42 
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